Leveraging a rolling cross-section design in panel studies to investigate police legitimacy
Monica Gerber and Cristóbal Moya
London School of Economics and Political Science
May 22, 2023
Today
Motivation
The rolling cross-section design
This Study: a rolling cross-section panel design
Implementation challenges
Motivation
Panel studies
Confirmatory research
Individuals are interviewed at two or more points in time
Possible to make inferences about changes in attitudes or behaviours at the individual level
Motivation
Panel studies
Confirmatory research
Individuals are interviewed at two or more points in time
Possible to make inferences about changes in attitudes or behaviours at the individual level
Rolling cross-sectional design (RCS)
Exploratory research
Cross-section: Data obtained at a specific point in time
RCS: distribute interviewing within a cross-section in a controlled (random) way over time
The rolling cross-section design
Unexpected event during survey design
Unexpected event during survey design (UESD): “research design that exploits the occurrence of an unexpected event during the fieldwork of a public opinion survey to estimate its causal effect on a relevant outcome by comparing responses of the individuals interviewed before the event \(t_i<t_e\) (control group) to those of respondents interviewed after the event \(t_i>t_e\) (treatment group).” (Muñoz et al., 2020, p. 187)
Effects of naturally occuring events that cannot be manipulated through controlled experiments (external validity)
Assumption that interview time is independent from the time when the event occurs
The rolling cross-section design
The “expected-unexpected” event during survey design
Temporal heterogeneity as an opportunity to move closer to causal inference (Johnston & Brady, 2002)
Steps:
Total sample is divided into multiple replicates (smaller random samples)
Replicates are assigned to be interviewed at different intervals of times
By itself it is not able to capture individual change
The rolling cross-section design
Assumptions (1)
Temporal ignorability: “for any individual, the potential outcomes must be independent from the moment of the interview” (Muñoz et al., p. 189)
Moment of the interview should be as good as random
In RCS moment of interview is random
Protocol to ensure equal procedures to contact and recontact sample units
Reachability: ease of contact may vary between respondents
Attrition: nonresponse may be correlated with potential outcomes
The rolling cross-section design
Assumptions (2)
Excludability: “any difference between respondents interviewed before and after the event shall be the sole consequence of the event” (Muñoz et al., p. 189)
Timing of the interview \(t\) affects \(Y\) only through event \(T\)
Beware of collateral events, simultaneous events and unrelated time trends
Need to characterize the event and its reactions
The rolling cross-section design
Assumptions (3)
Compliance: “some of the subjects were not really exposed to the event”Assignment to the treatment group \(t_i>t_e\) was assumed to perfectly correspond to actually receiving treatment \(T=1\).” (Muñoz et al., 2020, p. 194)
Need to measure compliance
The rolling cross-section design
Example
This Study
Police legitimacy
Police work requires public support to be able to function in a democratic context
Beliefs that the police act in procedurally just ways and are legitimate facilitate compliance and cooperation with authorities
A socially legitimate police force should require little violence to confront opposition
However, the world has seen many cases of violent demonstrations, police violence and human rights violations
This Study
The Chilean case
Chile’s main police force, the Carabineros, was among the most trusted institutions (Dammert, 2019)
Yet, recent events of corruption and excessive use of force during the social uprising in October 2019 have produced a serious loss of trust
Crisis of legitimacy: proposals of “deep reform” and even the dissolution of the institution
Need to understand relationship between police and citizens in Chile during the next crucial years
This Study
A rolling cross-section panel design
Probability-based online panel of the Chilean population (18-65 years old)
n = 5.300 in Contact Survey
Implementation challenges
Narrower bandwidths:
Will reduce N and statistical power
Can reduce bias in the presence of collateral events or confounders correlated with time of the interview
Can increase management costs
Interview window: for how long should respondents have the chance to complete the survey?